Expose --num-workers and --native-parser#21387
Conversation
This comment has been minimized.
This comment has been minimized.
JukkaL
left a comment
JukkaL
left a comment
There was a problem hiding this comment.
Thanks, left a few minor comments.
| Parallel type-checking | ||
| ********************** | ||
|
|
||
| By default, mypy checks all modules in the same Python process, this can be slow |
There was a problem hiding this comment.
Grammar nit: '... process. This can be slow ...'
|
|
||
| * Specifying a number of workers that is larger than the number of *physical* | ||
| CPU cores is not beneficial, since mypy is usually CPU bound. Best way to | ||
| tune the number of workers on a given machine is to start from 2-3 workers |
There was a problem hiding this comment.
Maybe suggest starting from 4 workers, since almost all CPUs have at least 4 cores these day (except for some CI jobs perhaps)?
There was a problem hiding this comment.
Reasons I think 2-3 is a good start:
- We need to save one core for the coordinator process.
- GitHub action runners are usually really basic.
Anyway, I can probably say "3-4" instead of "2-3".
| Use ``NUMBER`` parallel worker processes (in addition to the coordinator | ||
| process) to perform type-checking. Specifying ``--num-workers 0`` (default) | ||
| disables parallel checking. Automatic detection of the optimal number | ||
| of workers is not supported yet. |
There was a problem hiding this comment.
Do we limit the number of workers based on the number of actual cores, e.g. if only 4 cores are available and 16 workers are requested, are we still using 16 workers?
There was a problem hiding this comment.
No we do not. Actually during parallel parsing we even do the opposite: if the auto-detected number of cores is smaller than --num-workers, then we use --num-workers instead.
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Three things here: * Add the docs for `--num-workers` and `--native-parser` * Remove `argparse.SUPPRESS` from them * Add couple checks for `num_workers`
2 participants
Three things here:
--num-workersand--native-parserargparse.SUPPRESSfrom themnum_workerscc @JukkaL